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ABSTRACT 

The OSAP High Risk Youth Demonstration Program of the 
Mississippi Band of Choctaw Indians is directed toward the prevention 
of substance abuse through experiences offered in an after-school 
program. This is the second year that the program has been 
implemented in the six elementary schools and one middle school of 
the Mississippi Choctaw reservation. In 1990*91, the project was 
designed to serve an estimated 610 students, although 710 were 
actually enrolled. The program included five components: (1) tutoring 
on homework; (2) drug education; (3) recreation; (4) snacks; and (5) 
cultural activities. The external evaluation included a review of 
activities, verification of data collection for internal evaluation, 
a review of the preceding evaluation, a summary of evaluator field 
visits, including interviews with teachers, principals, and parents 
in participating schools, and a report of observed strengths and 
weaknesses. In five of the seven schools, the program is being 
implemented with great fidelity to the model described in the funding 
proposal. Overall, the program is very effective. Although community 
substance abuse has not lessened, very few cases are found among the 
children. Parent and student attitudes toward the program are 
favorable. It is also apparent that a strong principal is the key to 
effective program functioning. (SLD) 
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External Evaluation of the Mississippi Band of Choctaw Indians' 
OSAP High Risk Youth Demonstration Program _ Year Two 

M. Mas Brown 
Mississippi Band of Choctaw Indians 
Jimmie C. Fortune 
Abbot L. Packard 
Virginia Tech University 

Introduction 

The Mississippi Band of Choctaw Indians' OSAP High Risk 
Youth Demonstration Program is directed toward the prevention of 
substance abuse through experiences offered in an after- school 
program. This is the second year that the program has been 
implemented in seven of the eight reservation schools, the six 
elementary schools and the middle school. The first year of this 
program allowed much of the implementation to be left up to each 
individual participating school. This effort was judged highly 
successful in that the project design permitted individual school 
programs to be tailored to the needs and differences in each 
school • 

In 1990-91 the project was designed to serve an estimated 
group of 610 kindergarten through eighth grade students. The 
project, however, was so popular that year that 710 students 
enrolled and were served. The 1991-1992 program had at the time 
of the external evaluator's visit in mid May, 709 students 
enrolled. The general overall structure of the 1991-1992 project 
included five components common to all participating schools and 
most enrolled students. These components included: tutoring on 
homework; drug education; recreation; snacks; and cultural 
activities. All school programs were held after school four days 
per week, allowing the programs to serve a "latch-key" function 
that is liked by many parents. Many students expressed the idea 
that the program was M fun. M Others benefited from the tutoring 
associated "homework", a component enjoyed by many parents of the 
older children. Still other children enjoyed the program because 
of the recreation and cultural activities. Hence, the project 
contained programs that included something to be liked by 
everyone. The notion of "responsive programming" appears to be a 
key in successful prevention programs. Certainly it is a key to 
the success of this one. 

External Evaluator Responsibilities and Methodology 

The responsibility of the external evaluator includes the 
verification of the quality control and operations components, 
and the advisory oversight of the internal evaluation. This 
external evaluation to the OSAP "after- school" program included 

the following: 

1. A review of records and project activities; 

2. A verification of the collection of data to be used 
in the internal evaluation of the project; 

3. A review of the 1990-1991 final evaluation results; 

4. A summary of evaluator field visits to each school; 

5. A report of observed strengths and weaknesses; and 



7. Recommendations for future management action. 
Many of the external evaluator activities involved reading 
archival data files, seeing if proposed data collection was 
taking place, verifying that reported project activities were 
actually occurring, and ethographically seeing if clients 
(parents and children) were satisfied with the project. The 
external evaluator sees if the project files contain what they 
are supposed to contain and sees if these files are being used as 
reported. Review of the collection of the evaluation data 
involved review of the instruments used to collect information 
and review of the methods through which the information was 
gathered and planned to be analyzed. The method used to verify 
project activities and to gather data on the field visits to each 
school was ethnographic interviews and the summary of results of 
reviews of the archives. From these methods follows the report 
on the strengths and weaknesses of the program; and the rendering 
of recommendations. The external evaluation efforts for the two 
years are similar except the second year involved only one 
project site visit. 

Accountability 

Records review . The external evaluator reviewed the records kept 
pertaining to program operation. These records included a 
program participation application forms for students; records of 
student attendance; records of student discipline referrals; 
staff rosters, menus of snacks; activity outlines collected 
weekly from the teachers in the schools; purchases of supplies, 
and other financial records. An audit trail has been developed 
for cost accounting. 

Basic operations information including attendance and 
personnel information for both years are clear, comprehensive and 
accurate. Less staff turnover and a much lower percentage of 
student dropouts late in the year are occurring the second year. 
The second year program features 90% retention of enrollees in 
all schools except the middle school where band, cheerleading and 
spring sports compete. In this school it was nearly a 70% 
retention rate. 

Verification q£ thft Evaluation Data . This external evaluator was 
able to review the 1990-1991 results of data analysis conducted 
by Mississippi State staff. The evaluation and data analysis 
staff at Mississippi State is to be complimented for their 
exemplary performance as internal evaluators, given the emergency 
assumption of this role in 1990-1991. Overall as external 
evaluator I was very pleased at the quality of this work. The 
few comments I have are directed toward fine-tuning the 1991-1992 
analysis. 

The results of the Choctaw Picture Self-Concept test appear 
promising and the development effort on this instrument should 
continue. This effort should be increased due to the importance 
of the self -concept variable in the prevention process and the 
"apparently invalid" performance of the Tennessee Self-Concept 
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Scale (TSCS) with the Indian children. Problem* with the TSCS 
are moat likely environmental or cultural in that the scale 
assumes social interactions not present in the reservation 

setting. 

The report on the Parent Interview Form (PIP) would be more 
helpful if presented by school. Response stubs on this 
instrument are "poor" and perhaps condescending. Due to the way 
parents tend to think about the word "poor" pertaining only to 
possession of wealth this seems to be an inappropriate term for a 
Likert scale and "ugh" may be "cute white society" terminology, 
but not Choctaw. Why use a Likert format if scaled scores are 
not to be analyzed? 

The same response stub descriptors were also used on the 
child interview forms, most likely biasing the results from the 
negative. The finding that an increasing satisfaction with 
snacks occurs commensurate with the timing of short-change of 
snacks last year is puzzling. No shortage of snacks were 
observed for this second year. 

The reported analysis of the Drug Education and Substance 
Abuse Form for 1990-1991 was very difficult to read and 
understand without sample cell sizes and percentages of response 
by stub. It appears that the drug education information was only 
partially effective. Teacher*' remarks in two schools for the 
year 1991-1992 suggest better use of clearing house materials 
should be made. 

Generally, I agree with the school dropout report, but there 
are a few misleading statements in the report. First, the out- 
migration from the reservation may not be consider minimum in 
many respects although the out-of-state migration is minimum. 
Over the period 1986 to 1990 in the Demographic Survey 97 
students have been identified as leaving the reservation setting 
or leaving the reservation schools. Some of the 172 out-of- 
school youth reported in 1986 returned to school prior to the 
1990 demographic survey. The dimension created by the 
maintenance of the dormitory is associated with dropouts, as well 
as the number who quit high school to work in tribal enterprises 
and attend adult education. The Choctaw dropout is a more 
complex problem that reflected in the report. The OSAP After 
School appears to have great potential in addressing the dropout 
problem. 

Implementation 

As part of the eummative external evaluation for the program 
year 1990-1991, visits were made to each school and school 
personnel and after-school program personnel were interviewed . 
In some schools classes were observed and children were briefly 
interviewed. In all schools the principal was interviewed. In 
the 1991-1992 program year all schools were visited again by the 
external evaluator. The evaluator was able to attend one program 
social, a cook-out, and was able to observe classes in four 
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school*. Teachers, aides, parents and students were interviewed 
in four schools. Teachers, aides and parents were interviewed in 
a fifth school. Teachers and aides were interviewed in a sixth 
school. Six of the seven principals were interviewed. 

Field Notes from School Visits 

Standing Pine - The 1990-1991 report for Standing Pine went as 
follows. I talked to the principal, two aides and two teachers 
concerning the program. The program involves 52 children divided 
into four groups. The principal reports that the program is 
meeting a need in the community. All four group leaders (the two 
aides and the two teachers) were extreuely positive about the 
program. All five meet as a group, appeared a little reluctant 
to talk until they had to answer a couple opener questions. The 
four teachers rated the program a-, a, a, and a. Most 
outstanding accomplishments of the program are: the beadwork of 
the students, student participation in dances and the opportunity 
for the teachers to get the students started right with their 
homework. The drug- free instruction is understood and believed 
effective. Drug problems are nearly non-existent with the 
children and about 50% of the families have alcohol problems. 
Problems include: for two months in the winter some children 
arrive home by bus after dark (so far no problems from this and 
it is thought to be the lessor of two evils, children arriving 
home without a supervisor present) , craftsmen have to travel so 
far to the school that they want mileage paid, teachers have 
difficulty in obtaining "the right beads" (can not be purchased 
in the local area and teachers missed the purchasing opportunity) 
and teachers want feedback from tests and sociogram. 

The 1991-1992 report for Standing Pine is as follows. I 
talked to the principal, two aides and three teachers concerning 
the program. The program involves 52 children divided into four 
groups. The principal reports that the program is meeting a need 
in the community. All four group leaders (the two aides and the 
two teachers) were extremely positive about the program. The 
staff meet as a group. The teachers and aides rated the program 
a-, a-, a, a, and a. Child-related drug problems are nearly non- 
existent, but nearly half of the families have alcohol problems. 
The transportation problems have been solved by taking the eight 
students not participating home in a van. Much better use of 
community resources and craftsmen are being made. School 
projects have focused on traditional dress. The problem with the 
"right -beads" persists but is less. 

Red Water - In 1990-1991 project year, 1 talked first to the 
principal and then talked to three aides and one teacher while 
students were doing a cultural dance. The program has between 
58-60 students, two teachers and three aides. The outstanding 
characteristics of the program were: cultural dancing and 
beadwork. It appears that the principal has not been supportive 
of the after school program during the school year. In fact, the 
principal stated that she felt that the program confused the 
children and they could not distinguish the program from regular 



school. She felt that lack of discipline in the program each day 
was carrying over to the next morning (children will fight in 
after school program and then open up hostilities again in the 
morning) . Teacher and aides all believe that the program has 
been a success, especially with cultural instruction. Problems 
noted by the teacher included: homework should be done at home 
(the homework part simply turns into more school work) ; the 
program needs more community support and parent participation 
(have trouble getting parents to spend their time at the time 
that the program is in operation) ; need beads and more time to 
complete teaching beadwork; need boards for instruction on making 
"God' s Eye" craft; and there is a problem the way that the 
children enter the single room (lunchroom) . Teachers simply turn 
their students out and the students arrive at the program at 
different times. Although drug-prevention instruction is 
provided, they feel that drinking among parents may be on the 
upswing. No substance abuse among the children, 60 to 65% 
parents have alcohol problems. 

In 1991-1992 project year, I talked first to the new 
principal and then talked to three aides and two teachers. I 
observed two classes, interviewed a dozen students and two 
parents. The new principal has done a "fantastic" job renewing 
community interest in the school. In one of the classes that I 
observed, paper flowers for Memorial Day activities were being 
made. The students were doing a near professional job in a 
relaxed setting. Their taste in color combination is unique and 
good. A few students by choice in the class were doing beadwork. 
In the second supervised sports participation was taking place in 
the gymnasium. 

The new principal is working on a plan to support locally 
the extension of the program to 5 p.m. to support parents wishes. 
Currently, the program runs until 4:30 p.m. prior to the children 
being bused home. No transportation problems, snack shortages, 
materials problems or vandalism were reported. Teachers rotate 
children and report that they are not as tired as last year at 
this time. No substance abuse among the children, about 60 of 
the parents have alcohol problems. 

Conehatta - In 1990-1991 project year, I talked to the principal, 
the lead- teacher and four aides. The outstanding achievements of 
the program was that it provided the students with another hour 
and half of positive reinforcement and provided them with an 
opportunity to learn traditions of the culture. The program 
included 112 students, 4 teachers and 6 aides. The program is 
really needed because most of the parents work. The principal 
felt that the two months of dark to and from school needed to be 
studied. He felt that the program put pressure on the school 
transportation system. He was unaware that the program 
contributed to paying for the school transportation required by 
the program. He also pointed out that the program caused him to 
have to have two faculty meetings - one at 2*00 p.m. for the 
program teachers and one at 3s 30 p.m. for the other teachers. 
The aides participating felt that the program needed t more beads 



(invoicing took too long and they did not know what to anticipate 
as program needs) ; planned activities that permit the children to 
be outside; and more structure to what happens in the program. 
The lead teacher said that the program needs: a one hour per week 
planning time, feedback on the sociogram and more program 
guidance. Tobacco use is a major problem with a majority of the 
older students. Alcohol abuse in 70-75% of the homes. They need 
a list of resource people and some help in the substance abuse 
instruction. They feel that the homework tutoring is "good" and 
is "working" , but they need better coordination on assignments • 

In 1991-1992 project year, I talked first to the principal 
and then talked to three aides and two teachers. I observed two 
classes, interviewed a dozen students and one parent. The after 
school participants are divided into six classes of 12-15 
students headed by one teacher and an aide. The beadwork 
observed in one class was excellent. In the second class study 
time was in session. Students were very satisfied with the 
program. The teachers reported a need for more drug education 
materials. They report that "videos" and workbooks are needed as 
written materials do not seem as effective as those requested. 
Snacks and beads were ample this year. Emphasis is still being 
placed on "outside" experiences such as traditional sports and 
homework tutoring. Tobacco use remains as a major problem with 
older students and adults. Alcohol abuse remains high, but has 
plateaued. 

Tucker - In 1990-1991 project year, I talked with the principal, 
the lead- teacher, two aides and one teacher not involved in the 
after school program. The program has three teachers, three 
aides and about 110 students. The principal sees as strengths of 
the program the following: students have become more involved 
with cultural activities; the tutoring has helped get students 
doing homework on a more consistent basis; and the program has 
helped many parents by serving a latch key function. Problems 
include: the principal has had to put in some overtime for which 
she did not receive pay. There have been no discipline problems 
associated with the program. The lead- teacher believed the 
accomplishments of the program to include the following: the 
students benefit from the physical exercise, social interaction 
and cultural activities. During the year in the after school 
progr am the students have designed t- shirts, done beadwork, 
received instruction on the losses due to drug abuse and 
developed homework habits. The students have fun and the 
teachers switch program parts to keep the children's interest. 
The teacher rates the program at Tucker as a B. This is because 
she kept a group as lead- teacher for the first half of the year. 
In the second half of the year the lead- teacher floated and 
stimulated the groups. This was a better arrangement. She saw 
one major problem: the lead-teacher has no authority or power 
with staff or students. The two aides thought the program to be 
"good" because culture is being taught and the children and 
parents like it. The problem that they see is the lack of 
materials to make cultural artifacts. The other teacher wished 
for more tutoring time for the older children and science to be 
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taught in the program. The program did not have a bead shortage. 

In 1991-1992 project year, I talked first to the acting 
principal and then talked to three aides and three teachers. I 
interviewed a half-dozen students and one parent. Tucker's 
program is in a transitional stage. The program will finish with 
78 students, but the two group structure has been made complex by 
some recent decisions. Throughout the year the program operated 
with a three group design. Two groups, one for elementary 
students and one for primary students, were offered a program of 
arts and crafts, recreation and sports, drug and substance abuse 
education, snacks and some homework tutoring. Each of these 
groups numbered about 18 students each. A third group of 42 
upper grade students were offered a program of snacks, drug and 
substance education, extended homework tutoring, and brief 
recreation and play after homework was completed. It was 
discovered that some of the older students were lying about 
completing their homework, so play privileges were removed from 
the group of 42, resulting in some of the older students going to 
the crafts group. Some problems have been reported with regard 
to students taking the early bus without permission. Program 
features some additional counseling of students. 

Bogus Chitto - In 1990-1991 project year, I talked to the 
principal, two teachers and two aides. The lead- teacher was not 
available. The principal has tried to blend the activities of 
the after school program with the SAVE THE CHILD program and with 
local music instruction (choir, piano and guitar) . The program 
has served a valuable latch key function and the homework part 
has served to increase achievement. The after school program has 
four teacher positions filled by five teachers and five aides. 
The program serves 110 children. The program has caused no 
discipline problems but has made the principal use the activity 
for faculty meetings and has made each meeting require two 
sessions. The principal feels that the program at Bogue Chitto 
needs one additional teacher and additional resource money. The 
two teachers rate the program as a B+. Both teachers praised the 
program in several ways. The program offers an opportunity for 
the children to have fun at school, to study their culture and to 
learn study habits by starting their homework. The aides enjoy 
teaching the physical activities and the cultural activities. 
They wish to be allowed to work the same number of hours as the 
teachers. Both aide and teacher meet with rather large subgroups 
(four) • 

In 1991-1992 project year, I talked to the two teachers, an 
aide and one parent. The principal was away on a field trip and 
was interviewed later by telephone. The program has 115 
participants, five teachers and five aides. It was reported by 
the lead teacher and later confirmed by the principal that no 
problems were encountered by transportation and that the program 
ran very smoothly with regard to administration. The program 
featured 10 groups of students, each under the supervision of one 
staff member. Student groups were rotated across supervisors so 
that everyone got to know each other. Each group received music, 
drug and substance abuse education, cultural arts, recreation, 
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snacks and tutoring. The eighth students gained experience as 
aides with the younger groups. This worked out very well. The 
students made t- shirts and beadwork (necklaces/ rings or key 
chains) . 

The teachers would be happier with the project, if they 
could buy their own supplies. Drug education and substance abuse 
materials appropriate for young children .were short. Snacks were 
liked much better this and no troubles with transportation were 
reported. Alcohol abuse remains a major social problem in the 
community. 

Choctaw Central , grades 7 & 8 - In 1990-1991 project year, I 
talked to the principal, two teachers and two aides. I talked 
with the assistant principal and the lead- teacher. The assistant 
principal's reaction to the program was "Great Program". He felt 
that the key product of the program was the tutoring and the help 
with homework. The other beneficial products were the latch key 
function served to working parents and the provision of exercise 
and physical education to the students. The program includes 
three teachers, two aides and 44 children. He would like to see 
the funding for materials increased and he would like to increase 
the time spent tutoring. There are two regular program groups 
and one group that is all study hall by parent request. The 
lead-teacher enjoyed the program and will do it again. He rates 
the program an A and reports that the students really like it. 
He believes that the program is good because of the teaching of 
culture and tutoring. 

In 1991-1992 project year, I talked to the principal, one 
one aide and six parents. The program has two teachers, five 
aides and 28 remaining students, although at different times of 
the year up to 60 students have participated. It must be kept in 
mind that the program has to compete with several traditional 
school activities, such as junior varsity sports, cheerleading, 
band, and chorus. Parents expect this program to serve as a 
tutoring program to address the academic difficulties of the 
students. The principal is most happy with the program as it is 
currently configured. He is looking for a way to promote 
academics, but not to focus solely on homework. The "Great 
Books" reading program was discussed as a potential one day a 
week activity. 

Pearl River - In 1990-1991 project year, I talked to the 
principal, one staff member and the lead- teacher. The program can 
be credited with the following achievements: it reinforces 
homework, it conveys the drug education message, it provides an 
opportunity for the students to do some needed physical 
education, and it sets the opportunity for the children to 
socialise. The program involves seven teachers, seven aides, and 
261 children arranged in six groups. The lead- teacher £loats and 
organizes. The program negatives include: staff fatigue due to 
oversubscription, facility deterioration, lack of control and 
supervision by program administration, and poor quality snacks. 
It is suspected that snacks are leftovers, but they should not 



be. The lead- teacher reported difficulty in keeping staff. Some 
have reduced their time in half, putting pressure on him to keep 
the needed number of staff. With the size of the program someone 
is needed to answer the telephone. The floating method works 
better for the lead- teacher. He gave the program a B. 

In 1991-1992 project year, I talked to the principal, four 
teachers including the lead teacher, three aides, fourteen 
students and eighteen parents at the cookout and after school 
program, occurring on the visit day. The program has seven 
teachers, seven aides and 270 remaining students. The principal 
reports that the program has been a real pleasure and has run 
smoothly. The teachers are very pleased with the administrative 
support, but feel that approximately forty children in a group is 
too many. The students like the crafts, music, and recreation 
time. Many of the parents are pleased with the culturally 
related instruction, the crafts, the music and the latch-key 
functions served by the program. The tutoring start given for 
homework is welcomed by both teachers and parents. Snacks, play, 
sports and socializing are enjoyed by the students. Overall, the 
program is big, well-liked, and making a true contribution. 
Staffing nor transportation have been big problems this year. 

Summary of Results 

In 1991-1992 project year, five of the seven schools the 
program appears to be implemented in "dream- like" fidelity to the 
model described in the funding proposal. In 1990-1991 only three 
schools were as well implemented. Again the keys to success in 
all five schools appear to be: a strong principal who is able to 
display confidence in the staff during the development and 
operation phases of the program; an energetic lead- teacher who 
coordinates efforts, meets deadlines and communicates; and staff 
that enjoy seeing students enjoy school, which could be a theme 
of the program. Support from parents in every community has been 
outstanding, but unfortunately alcohol abuse has not lessened in 
the communities. In one school where there is some difficulty 
with the program, the principal has been reassigned and the 
decision not to permit play needs to be re-evaluated. In the 
other school, age of child and competing middle school activities 
appear to reduce program impact by having a high degree of 
student migration both into and out of the program. In all seven 
of these schools there is strong evidence that the program is 
working on several dimensions. Such evidence includes: very few 
cases of substance abuse among the children, less gang activities 
and expressions of "good school attitudes." 

Overall, one would have to say that the program is very 
effective. Impact can be viewed as an absence of problems with 
students who are not supervised every afternoon. Gone are the 
problems with transportation, snacks, shortages of many 
materials, and teacher turn-over. All principals like and 
support the program. Students like the program. Parents like 
the program, but wish that it could give them a little more 
guarantee of academic success for their children. Teachers are 
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mora enthusiastic about working in tha program. 

Strengths and Weaknesses . It appears clear from the field notes 
for both years that the programs within each school have the 
strength of a good design shell. The common components of: 
tutoring on homework; drug education; recreation; snacks; and 
cultural activities are clearly the strengths of the programs. 
By making students feel closer to their culture, by having them 
enjoy school and by helping them with their homework, a true 
program of prevention is most likely operative. Individualized 
school level design appears to be both a strength and a weakness. 
The individualized programs are hard to manage from the project 
level perspective. By allowing each school to establish its own 
tailoring of the program, one gets a program that is highly 
effective at meeting the localized needs of each school. But one 
of the sacrifices made by individualization is the need to 
purchase materials for each school individually. A second 
sacrifice is the loss of uniformity and the ability to maximize 
overall project goals. In most schools the staff (especially the 
lead- teacher) are a strength. One must be careful not to allow 
the program to focus too much on homework or care of the facility 
or it will become more of the same rather than a preventive 
measure • 

The great need for the program and the general applicability 
of it to local social problems is worth pursuit. The second year 
of the project shows a large increase in program under standing 
and management. The program affords the opportunity for people 
to work together for a better future. 

Recommendations 

1. As I did last year, I heartily recommend that the program 
continue and that word of its success leak out to the rest of the 
world. 

2. Some insertion of teacher planning time should be made. 

3. There seems to be different levels at which programs work 
with homework and those that require students to do all of their 
homework do not work as well as those which make certain that 
students can do their homework. Emphasize the latter and 
discourage the former. For the most part this appears to be the 
predominant practice. 

4. Promote the "orderly fun" concept in the schools, no 
prevention will occur if the students do not like the program. 

5. A greater emphasis on reading the classics in literature 
may be good rainy day fund and may serve to develop greater 
educational maturity. 

6. More coordination is needed at two schools in the 
acquiring of clearinghouse materials on drug and alcohol abuse 
education. 
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